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Appendix A. About the study 


This appendix provides additional information about Georgia’s flexibility policy as well as relevant research and 
the context of the current study. 


Georgia’s district and school flexibility policy 

In 2007 state legislators passed an amendment to Georgia Code Title 20—the state’s education law—allowing all 
school districts to seek Charter System status, a title designated by state policymakers, by entering a performance 
contract with the state. A Charter System is a district that receives all allowable waivers of state law and 
regulations—the same flexibilities that individual charter schools in the state have been granted since their 
inception in 2000. State legislators passed another amendment in 2008 that gave local school districts two 
additional options. Districts could become Strategic Waivers School Systems (SWSSs), allowing them to request 
specific waivers in exchange for agreeing to meet school academic performance targets and face the loss of 
governance over schools otherwise, or Title 20/No Waivers School Systems, districts with neither waivers nor 
stipulated accountability. Districts that chose to become SWSSs could apply for all waivers that Charter Systems 
automatically received; the waivers had to be connected to improving student outcomes. Between 2008 and 2014 
only three of Georgia’s 180 districts applied to become an SWSS, and 28 applied to become a Charter System. 
However, all districts were required to select an option by June 30, 2015. As a result, by the end of the 2016/17 
school year, 178 districts—all but two—had adopted a performance contract, with 136 operating as an SWSS and 
42 operating as a Charter System. To date, no SWSSs or Charter Systems have faced consequences for failing to 
meet performance targets. This is in part because performance targets have been revised due to changes in 
standardized assessments and changes to Georgia’s College and Career Ready Performance Index. 


The state classifies the waivers available to both SWSSs and Charter Systems into three categories: academic, 
human resources, and financial. Academic waivers include flexibility from laws related to educational programs, 
curriculum and instructional courses, the organization of schools, promotion and retention, graduation, and 
attendance. Human resources waivers include flexibility from laws related to class size, personnel, certification, 
and professional learning. Financial waivers include flexibility from laws related to expenditure control, program 
appropriations, facility requirements, and funding formulas. Specific waivers within these categories include 
waivers that allow school districts the flexibility to increase class size, revise graduation requirements, and spend 
less than 65 percent on direct classroom expenditures, among other options (table A1). 
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Districts applying to become SWSSs were required to request at least one of the “big four” waivers, which cover 
class size and reporting requirements, teacher certification requirements, salary schedule requirements, and 
direct classroom expenditure control (noninstructional and instructional categorical allotments). SWSSs also could 
request waivers from other state regulations, except those related to student health and safety. SWSSs requested 
far fewer waivers than the total number of waivers allowed. Across all SWSS performance contracts the state 
issued waivers from 122 regulations. On average, each SWSS requested 39 waivers, and more than 5,000 waivers 
in total were granted across the 136 SWSS districts. The fewest number of waivers an SWSS requested was 22, 
and the most was 66 (correspondence with the Georgia Department of Education, 2019). 


Table A1. Examples of most commonly granted waivers from Georgia regulations 


Number of 
Strategic Waiver 


School Systems 


Waiver Relevant regulation with waiver 
Academic program flexibility 
High school graduation requirements 160-4-2-.48 136 
Early intervention program for students at risk §20-2-153 136 
Remedial education program §20-2-154 136 
Program for English learner students §20-2-156 133 
Promotion and retention §20-2-283 133 
Competencies and core curriculum §20-2-142 131 
Alternative education program §20-2-154.1 131 
Comprehensive health and physical education program plan 160-4-2-.12 129 
General and career education programs 20-2-151 129 
Online learning 20-2-140.1 124 
School climate management program §20-2-155 114 
Education program for gifted students 160-4-2-.38 112 
Human resources flexibility 
Organization of schools; employment of school administrative managers §20-2-290 136 
Personnel required 160-5-1-.22 135 
Class size and reporting requirements? §20-2-182 135 
Teacher certification requirements? §20-2-108, §20-2-200 135 
School day and year for students and employees §20-2-151, §20-2-160(a), 130 
§20-2-168(c) 
Salary schedule requirements? §20-2-212 126 
Financial flexibility 
Direct classroom expenditure control? §20-2-171 136 
Program weights to reflect funds for maintenance and operation of facilities §20-2-183 135 
Program weights to reflect funds for media specialists §20-2-184 135 
Program weights to reflect funds for salaries for assistant principals and secretaries §20-2-185 135 
Allocation of funds to pay beginning salaries of superintendents, secretaries, §20-2-186 135 
accountants, nurses, and certain other personnel 
Quality Basic Education funding formula §20-2-161 130 
Categorical allotment requirements (funding for direct instructional, media center, §20-2-167, §20-2-183 to 127 
staff development costs) and budget reporting §20-2-186 
Scheduling for instruction/program enrollment and appropriations §20-2-160 126 


a. “Big four” waiver. Districts applying to become Strategic Waiver School Systems were required to request at least one of the “big four” waivers. 
Source: Georgia Department of Education administrative records, 2019. 
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Relevant research and context of the current study 

Early research on school and district changes in response to Georgia’s flexibility policy found that staffing and 
spending at the district level increased after performance contract adoption; increased funding from the state 
was somewhat offset by decreased funding at the local level (Kramer et al., 2017). To date, however, no research 
has studied how those staffing and spending changes are associated with changes to other practices or to student 
achievement in Georgia districts that converted to an SWSS or Charter System. 


The results of this study also could inform similar policies in other states. Georgia’s flexibility policy provides local 
districts autonomy in the form of deregulation, or “autonomy from higher levels of government” and local control 
over decisionmaking (Finnigan, 2007, p. 505; Wohlstetter et al., 1995; Wohlstetter & Chau, 2004). Although 
Georgia’s system is unique among states in the extent of waivers granted, its application to all districts in the state, 
and the option to convert to a Charter System or SWSS (Kramer et al., 2017), other states, such as North Carolina 
and Texas, have undertaken efforts to increase public schools’ autonomy through flexibility. To date, little research 
is available on these types of policies to inform states interested in developing or improving similar policies or to 
help districts that participate in such policies implement waivers effectively. One of the few relevant studies 
currently available is Ladd and Zelli’s (2002) evaluation of North Carolina’s ABC’s program—a school-based 
accountability system that set expected growth standards and provided waivers from class size limits and teacher 
certification regulations as well as budgetary flexibility to help principals meet accountability targets. That study 
found that although the program led to many new practices aligned with state goals to increase student 
performance, it also might have encouraged high-quality teachers and principals to leave low-performing schools, 
which could increase inequality in the long term. 


Research on the governance model employed by charter school networks, which tend to implement more flexible 
governance structures than traditional school districts, is more readily available and is highly relevant to the 
Georgia context, where the charter school model has long been seen as a means to deregulation and increased 
autonomy for districts (Bulkley, 2005). Both the SWSS model and the Charter System model emphasize local 
governance and offer increased flexibility in return for greater accountability—key features of the theory of action 
for charter schools. The growing popularity of the charter school governance model has generated many recent 
large-scale education policy studies examining the relationship between how public education is structured and 
student outcomes. However, the results vary substantially (Clark et al., 2015; Gleason et al., 2010). For example, 
a research synthesis of experimental effect estimates of charter school enrollment on student achievement in 
reading and math identified some positive effects in elementary grades, but they were not consistent across 
subjects, grade levels, or locations (Betts & Tang, 2014, 2019). Other studies experimentally examined the effects 
of specific charter networks and found more consistently positive effects on student achievement, especially for 
traditionally disadvantaged students (Abdulkadiroglu et al. 2009; Abdulkadiroglu et al. 2011; Angrist et al. 2011; 
Angrist et al., 2010; Angrist et al., 2016). 


Given the mixed findings and limited information on whether and how districts and charter school organizations 
use flexibility to implement innovative practices, more research is needed to understand the conditions that 
facilitate implementation of waivers and new governance models, how waivers and new governance models 
encourage schools to implement new or innovative practices, and the relationship between those practices and 
student outcomes. 


In addition to informing the Georgia Department of Education’s efforts to support districts in implementing the 
state’s district and school flexibility policy, this study also contributes to the wider body of knowledge on flexibility 
policies, new or innovative practices, and the relationships between them. Although this study is specific to 
Georgia’s policy, elements of the policy, the waivers available, or districts’ experiences implementing innovative 
practices may be informative for other contexts. 
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Appendix B. Data and methods 


This appendix provides further information about the study’s data sources, sample, missing data, and methods. 


Data sources 


Data for this study are from four sources: Georgia Department of Education (GaDOE) administrative data, the U.S. 
Department of Education’s Common Core of Data, and survey and interview data collected by GaDOE and shared 
with the study team. 


Table B1. Student-, school-, and district-level variables used in the study 
WElarlel(=3 Description 


Source 


Gender Coded as female or male Georgia Department of Education 
Race/ethnicity Coded as American Indian/Pacific Islander, Asian, Black, 
Hispanic, multiracial, or White 
Special education Whether a student had an individualized education program 
status (IEP) 
English learner status Whether a student was identified as an English learner student 
Migrant status Whether a student is a migrant 
Grade level Indicates the student’s grade level, from 3 to 12 
English language arts Student standardized assessment scores on grade 3-8 English 


and math achievement language arts and math assessments and grade 9 English and 
grade 9-12 Algebra | end-of-course assessments for each year, 
grade, and subject available between 2005/06 and 2017/18 


School type Whether a school is a traditional public, public charter, magnet, Georgia Department of Education 
special education, alternative, or other school 

School Title | status Whether a school receives Title | funds as a school with a high- 
poverty population 

Other school-level Percentage of all students in a school in each racial/ethnic 

descriptors category 


Percentage of all students in a school who receive free lunch 
Percentage of all students in a school who receive reduced- 
price lunch 

Percentage of all students in a school identified as English 
learner students 

Percentage of all students in a school who have an IEP 


Strategic Waivers Coded as SWSS or Charter System Georgia Department of Education 
School System (SWSS) 

or Charter System 

District size Number of students enrolled in the district 


District urbanicity Coded as urban, suburban, town, or rural U.S. Department of Education’s 

Common Core of Data 
Performance contract Coded as early (2008/09-2014/15), middle (2015/16), or late Georgia Department of Education 
adoption cohort (2016/17) adopter 


Other district-level Percentage of all students in a district in each racial/ethnic 

demographic category 

characteristics Percentage of all students in a district identified as English 
learner students 


Percentage of all students in a district who have an IEP 

Percentage of all students in a district who are migrants 
Districts’ prioritized Whether a district indicated that an innovation was among its 
innovations top three priorities to implement 


Source: Administrative data from the Georgia Department of Education (GaDOE), publicly available data on district urbanicity from the U.S. Department of 
Education’s Common Core of Data, and data from surveys of district leaders conducted by GaDOE, 2019. 
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GaDOE administrative data. The study team established a memorandum of understanding with GaDOE that 
allowed the study team to use state administrative data to conduct the study. GaDOE provided data on every 
district’s status as a Strategic Waivers School System (SWSS), Charter System, or Title 20/No Waivers System, 
identifying the type of contractual flexibility agreement the district operated under and the year in which the 
performance contract began. GaDOE also provided data on student performance on state standardized exams 
and student demographic characteristics from 2006/07 to 2017/18 for the 178 districts that converted to an SWSS 
or Charter System. The study team aggregated student- and school-level data to the district level for each year for 
the following variables. 


Grade level. The percentage of students within the district in each grade level. 


Standardized English language arts and math achievement (grades 3-8). All Georgia students in grades 3-8 take 
an annual standardized end-of-grade assessment in the spring. The assessment that students took changed during 
the study period. The Criterion-Referenced Competency Tests were replaced by the Georgia Milestones 
Assessment beginning in the 2014/15 school year. To facilitate comparisons over time, the study team 
standardized students’ English language arts and math scores, using statewide means and standard deviations for 
students separately by grade level and year, to have a mean of O and a standard deviation of 1. This allowed 
examination of how students performed relative to all students in the state each year. 


Standardized English and Algebra | achievement (grades 9-12). The study team used grade 9 literature and 
composition as the end-of-course assessment in English. Georgia administers multiple end-of-course assessments 
in math. The study team used the most commonly administered of these, the Algebra | end-of-course assessment. 
Beginning in the 2014/15 school year, Georgia replaced its End of Course Test program with the Georgia 
Milestones End-of-Course Assessments system. The end-of-course assessments are administered at the 
completion of the course, regardless of students’ grade level, and serve as the final exam for the course. To 
facilitate comparisons over time, the study team standardized students’ end-of-course assessment scores, using 
statewide means and standard deviations separately by grade level and year, to have a mean of 0 and a standard 
deviation of 1. 


School type. Percentages of schools in a district each year coded as traditional public, public charter, magnet, 
special education, alternative, and other. Charter systems usually contain a mix of charter and traditional schools. 


Title |. Percentage of schools in a district each year that were eligible for Title | funding based on the concentration 
of students in poverty. 


SWSS or Charter System. Districts were identified as having made an initial switch from a traditional district to 
either an SWSS or Charter System based on their first performance contract with the state. Four districts changed 
from an SWSS to a Charter System in 2016/17 or 2017/18. These districts are classified as an SWSS because their 
initial contract was as an SWSS. 


Performance contract adoption cohort. The sample of 178 districts was categorized into three adoption cohort 
groups based on the year in which districts adopted a performance contract. The early adoption cohort group 
comprises the 3 SWSSs and 28 Charter Systems that adopted a performance contract between 2008/09 and 
2014/15, before Georgia passed an amendment requiring all districts to choose a status. The middle adoption 
cohort group comprises the 57 SWSSs and 4 Charter Systems that adopted a performance contract in 2015/16, 
immediately following the waiver amendment. The late adoption cohort group includes the 76 SWSSs and 10 
Charter Systems that adopted a performance contract in 2016/17, the last group of districts to do so. 


Gender. The study team aggregated student-level data on gender to calculate the percentage of students in a 
district who were male. 


REL 2021-051 B-2 


Race/ethnicity. The study team aggregated student-level data on students’ race/ethnicity to the district level to 
calculate the percentage of all students in a district who were in each racial/ethnic category. 


Individualized education program. The study team aggregated student-level data on students’ special education 
status to calculate the percentage of all students in a district who qualified for special education services and had 
an individualized education program. 


English learner status. The study team aggregated student-level data on students’ English learner status to 
calculate the percentage of all students in a district who were designated as English learner students. 


Eligibility for free or reduced-price lunch. GaDOE policy prevents the sharing of whether individual students are 
eligible for free or reduced-price lunch. Instead, the administrative data included a school-level measure of the 
percentage of all students in a school who were eligible for free or reduced-price lunch. The study team used this 
school-level variable to calculate the districtwide percentage of students who were eligible for free or reduced- 
price lunch. 


U.S. Department of Education’s Common Core of Data. The study team obtained data on school districts’ urbanicity 
(urban, suburban, town, or rural locale) from the publicly available Common Core of Data dataset for 2007/08. 


Survey data. GaDOE collected data from district leaders using a survey developed and programmed into 
SurveyGizmo by the study team and shared the results with the study team for the purpose of conducting this 
study (see appendix D for the survey instrument). GaDOE sent emails to superintendents in all 178 SWSSs and 
Charter Systems inviting them to complete the online survey using SurveyGizmo. The email also asked the 
superintendents to invite other key district staff, including assistant or deputy superintendents, chief financial 
officers, school improvement administrators, and local governance coordinators, to complete the survey. The goal 
was to receive at least one response from all 178 districts. The study team received responses from 133 districts 
for a 75 percent district-level response rate. A total of 183 completed surveys were received, reflecting multiple 
responses from 28 districts (16 had two responses, 6 had three responses, 6 had four responses, and 1 had five 
responses). When multiple responses were received, the study team selected one response to represent the 
district, with responses prioritized in the following order: superintendent, assistant superintendent, district chief 
of staff, and deputy superintendent. It is not known how many administrators received a request from their 
superintendent to complete the survey, so the study team could not calculate individual-level response rates. 


The key survey variables are the innovations that district leaders identified as their top, second, and third priorities 
after adopting a performance contract. District leaders were presented with a list of possible priority innovations 
and were asked to rank their top three priorities. The survey listed possible innovations in alphabetical order by 
type of innovation (with separate lists for academic, human resources, and financial innovations). The order of 
innovations on the survey has a .6 correlation with districts’ rankings of their top priorities. This could suggest that 
the order of innovations on the survey influenced district leaders’ rankings, but many of the innovations that 
GaDOE sensed districts most often prioritized were earlier in the alphabet (for example, alternate career 
pathways, alternative education programming, college and career academies, community-based learning, dual 
enrollment), and many of the innovations that GaDOE sensed districts have not prioritized were late in the 
alphabet (for example, school attendance, school counseling, school climate, student promotion, student 
retention, student placement, textbooks). Districts’ priority innovations should be interpreted with this potential 
limitation in mind. 


Interview data. GaDOE staff invited a subset of leaders who completed the survey to participate in a virtual 
interview using GoToMeeting software (see appendix E for the interview protocol). Using the analytic models 
estimating changes in student achievement after districts adopted a performance contract, the study team 
identified the top 15 percent of districts whose student achievement after adopting a performance contract most 
outperformed their student achievement before adopting a performance contract on each of the four state 
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standardized assessments included in the study: grade 3-8 English language arts, grade 3-8 math, grade 9 end- 
of-course assessment in English, and grade 9-12 end-of-course assessment in Algebra |. 


From among the 15 percent of districts that most outperformed their expected achievement in the first year after 
adopting a performance contract, GaDOE emailed leaders from 18 districts to invite them to complete an 
interview. These 18 districts were purposively selected because they had completed the survey and represented 
a variety of sizes, urbanicity, district types (Charter System or SWSS), and geographic regions of the state. GaDOE 
staff scheduled and completed interviews with administrators in 10 districts, with support from the study team, 
for an interview response rate of 56 percent. There may be some response bias in the interviewed districts, but 
districts were not selected to be a representative sample. Instead, GADOE was interested in interviewing leaders 
from districts that had experienced relatively large achievement growth after adopting a performance contract. 
The study team recorded all interviews and transcribed them using Rev.com. 


Sample 

The sample for the study included all 178 SWSSs and Charter Systems in Georgia. The sample for the analyses of 
district changes in student achievement after adopting a performance contract included students in grades 3-12 
from 2007/08 to 2017/18 in the 178 districts (table B2). The sample for the analyses of the association between 
changes in student achievement and changes in district practices included students in grades 3-12 from 2007/08 
to 2017/18 in the 133 districts for which a completed survey was received. The samples for the descriptive and 
qualitative analyses of district changes in practices included the 133 districts with completed surveys and the 
subsample of 10 districts with completed interviews. 


Table B2. Number of students and districts included in analyses of changes in student achievement 
Number of (eT g-lo(-Be) Grade 9-12 


years before (CT g-o(- Meiers} Grade 3-8 English end-of-course Algebra | end-of-course 
rolaryitele English language arts math assessment assessment 
performance 
contract 
adoption Students Districts Students Districts Students Districts Students Districts 
—10 218,371 72 218,937 72 38,170 71 27,533 71 
-9 504,392 146 505,326 146 85,371 143 69,596 144 
—8 568,759 156 569,917 156 97,179 154 50,125 83 
—7 576,356 159 577,495 159 96,179 157 47,218 84 
—6 618,655 161 619,007 161 102,886 160 91,615 150 
5 626,204 167 625,215 167 106,279 166 100,088 162 
—4 633,391 172 631,907 172 109,271 171 96,888 168 
-3 646,625 173 644,998 173 112,958 173 101,251 167 
—2 731,703 178 731,197 178 124,967 178 119,416 173 
-1 742,350 178 741,880 178 126,641 178 96,508 162 
0 752,816 178 744,446 178 127,429 178 79,163 146 
1 757,735 178 747,177 178 127,213 177 94,839 153 
2 534,705 106 527,931 106 90,696 106 75,764 95 
3 239,705 32 236,653 32 41,906 32 39,282 30 
4 178,851 22 177,284 22 31,002 22 28,614 21 
5 174,733 19 173,471 19 29,749 19 28,812 17 
6 136,168 17 135,897 17 22,732 17 20,769 16 
7 124,049 11 118,061 11 21,205 11 17,188 10 
8 115,187 6 107,216 6 19,800 6 15,213 6 
9 93,932 5 85,484 5 16,234 5 10,550 5 


Source: Administrative data from the Georgia Department of Education, 2006/07 to 2017/18. 
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Missing data 

The study team received responses from 133 of the 178 SWSSs and Charter Systems in Georgia. To test whether 
the sample of SWSSs and Charter Systems with survey responses is representative of all SWSSs and Charter 
Systems, the study team compared the two samples on student demographic and district characteristics. To do 
so, the team calculated descriptive statistics for each key variable and subtracted the value for all districts from 
the value for the sample of survey respondents. The raw differences between groups were converted to effect 
sizes reported as standard deviations (effect sizes are reported separately for the elementary and high school 
samples in table B3). The two samples differed by .05 standard deviation or more on only a few variables: Districts 
that responded to the survey were less likely to be located in rural areas and more likely to be located in suburban 
areas, were more likely to be an SWSS than a Charter System, and were more likely to be part of the early adoption 
cohort group than the middle adoption cohort group compared with districts that did not respond to the survey 
(see table B3). The distribution across performance contract adoption years also varies somewhat. However, these 
variables do not correlate strongly with the prioritized innovations. It is possible that despite the similarity of the 
two samples on these variables, the samples may differ more in their survey responses. 


Table B3. Respondents to the district survey are generally similar to the full population on observed 
characteristics 


Elementary school grades High school grades 


Survey Survey 
NFVar-le)(=1 PN] el isi da oansy respondents Effect size All districts respondents Effect size 


Test scores (student-level means standardized by grade and year) 


Grade 3-8 English language arts —0.02 —-0.01 —0.01 na na na 
Grade 3-8 math —0.07 —0.07 —0.00 na na na 
English end-of-course na na na 0.01 0.02 -0.01 
assessment 
Algebra | end-of-course na na na 0.00 —0.00 0.00 
assessment 


Other covariates (proportions) 


English learner student 0.09 0.09 0.00 0.07 0.07 0.00 
American Indian/Pacific Islander 0.00 0.00 0.00 0.00 0.00 0.00 
Asian 0.04 0.04 0.00 0.04 0.04 0.00 
Black 0.37 0.38 —0.03 0.39 0.39 0.00 
Hispanic 0.13 0.13 0.00 0.12 0.12 0.00 
Multiracial 0.03 0.03 0.00 0.03 0.03 0.00 
White 0.43 0.42 0.02 0.43 0.42 0.02 
Male 0.51 0.51 0.00 0.52 0.52 0.00 
Individualized education 0.11 0.11 0.00 0.10 0.11 —0.06 
program 

Migrant 0.00 0.00 0.00 0.00 0.00 0.00 
Charter waiver (district) 0.17 0.16 0.04 0.16 0.16 0.00 
Strategic Waivers School System 0.82 0.84 —0.09 0.83 0.84 —0.04 
waiver (district) 

Rural (district) 0.29 0.25 0.12 0.28 0.24 0.13 
City (district) 0.17 0.17 0.00 0.16 0.16 0.00 
Suburb (district) 0.46 0.50 -0.10 0.48 0.53 -0.12 
Town (district) 0.08 0.08 0.00 0.08 0.07 0.09 
Early adoption cohort group 0.32 0.35 —0.08 0.34 0.37 —-0.08 
(district) 
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Elementary school grades High school grades 


Survey Survey 
NV F-Var-le)(=1 PN] el isi da eansy respondents Effect size All districts respondents Effect size 
Middle adoption cohort group 0.39 0.35 0.10 0.38 0.34 0.11 
(district) 
Late adoption cohort group 0.29 0.30 -0.03 0.29 0.29 0.00 
(district) 
Performance contract adoption year 
2008/09 0.03 0.02 0.25 0.03 0.02 0.25 
2009/10 0.01 0.01 0.00 0.01 0.01 0.00 
2010/11 0.03 0.03 0.00 0.03 0.03 0.00 
2011/12 0.03 0.05 —0.32 0.03 0.05 —0.32 
2012/13 0.01 0.02 —0.43 0.01 0.02 —0.43 
2013/14 0.02 0.01 0.43 0.02 0.01 0.43 
2014/15 0.06 0.05 0.12 0.06 0.05 0.12 
2015/16 0.42 0.41 0.02 0.42 0.41 0.02 
2016/17 0.40 0.42 —0.05 0.40 0.42 -0.05 
Free or reduced-price lunch 0.60 0.59 0.03 0.51 0.51 0.00 
(school) 
Free lunch (school) 0.52 0.52 0.00 0.45 0.44 0.02 
Reduced-price lunch (school) 0.07 0.07 0.00 0.07 0.07 0.00 
Number of student-by-year 9,021,221 7,750,957 na 1,889,747 1,690,390 na 


observations 


na is not applicable. 

Note: Test score variables were standardized by subtracting the mean and dividing by the standard deviation within each grade and year. Effect sizes were 
calculated by subtracting the two test score values, as test scores already were standardized. Effect sizes for other variables were calculated by converting 
to log odds ratios and using the Cox transformation to the standardized mean difference scale, following What Works Clearinghouse Version 4.1 procedures 
(What Works Clearinghouse, 2020). 

Source: Authors’ analysis of administrative data from the Georgia Department of Education and data from the U.S. Department of Education’s Common Core 
of Data. 


Methods 


Quantitative analyses. The study team conducted a generalized difference in differences analysis. The analytic 
models used to estimate overall mean change in achievement after performance contract adoption (research 
question 1) took the following form: 


Yet = Po + BixPosty, + B2SY; + B3D_ + Ba Gur + BsZxe + 214 POSt et + ext, 


where Y;, is the average standardized achievement outcome (standardized scaled achievement score in English 
language arts, math, grade 9 English, or Algebra | for district k at time t), Posty; is a binary indicator signifying 
that district k has access to an approved waiver at time t (that is, it is set to 1 for all years that correspond to a 
performance contract that made waivers available and 0 otherwise), SY; is a vector of school-year indicators (from 
2006/07 to 2017/18), D; is a vector of district indicators, G,; is a vector of aggregate grade indicators (that is, the 
percentage of students in a grade in a district in a year) that correspond to the student testing groups for outcome 
Yee, and Z;; is a vector of time-varying district-level aggregates of student and school characteristics, including 
gender, race/ethnicity, individuated education program status, English learner status, eligibility for free or 
reduced-price lunch, and school type (that is, Title | status, alternative school, special education school, magnet 
school, charter school, or other school). The model allows £,;,to vary at the district level (that is, each district has 
its own estimate of relative change in achievement after performance contract adoption). The model has error 
term Z1;, which reflects district-level deviation in 8,, from the grand mean, $1, and ez, which is the residual error 
term. The focal parameters from this model are 6,, and the standard deviation of Z;;, T,, which quantifies the 


REL 2021-051 B-6 


variability in changes in achievement after performance contract adoption. This approach is similar to the 
approach developed by Raudenbush and Bloom (2015) and Bloom et al. (2017) for studying variation in site-level 
program effects. 


The study team used the estimates of t,, along with the mean [3,, estimates, to compute the percentage of 
districts that expected to have true changes of .05 standard deviation or more using the a cumulative normal 
distribution, evaluated at .05: 1 — Og, ,, (.05). Similarly, T,, along with the mean 3, also were used to construct 
90 percent prediction intervals (for example, Borenstein et al., 2011; IntHout et al., 2016). Importantly, results 
presented in this report are rounded, but the analyses were based on all significant digits. This may result in minor 
differences in manual computations of some table results. 


The model for research question 1 was expanded to examine the relationship between Post,; and district-level 
characteristics (research question 1) using: 


Yer = Bo + PixPosty: + B2SY; + B3D_ + Ba Gue + Bs Zxt + Boe Wy POSt yt + Z1~POSt KL + ext, 


where W,Post,; is a vector of interactions between Post,;, and district-level characteristics. For research 
question 1, Post,,; was interacted with urbanicity (urban, town, or rural), district type (SWSS or Charter System), 
and adoption cohort group (early, middle, or late). 


For research question 3, W,Post,, was expanded to included interactions with select’ priority innovations 
indicated by district leader responses to the survey. The main effects for W, are unnecessary in this model 
because it includes district indicators, D,. 


When evaluating the results of the analyses, the study team defined meaningful differences in changes in 
achievement as differences that are statistically significant and .05 standard deviation or larger. The study team 
defined small differences in changes in achievement as those that are statistically significant and smaller than .05 
standard deviation. Empirical benchmarks of annual student growth (Hill et al., 2008) translate .05 standard 
deviation to approximately 5 and 6 weeks’ worth of total achievement gain (year over year) in grade 3-8 math 
and English language arts, respectively, assuming 36 weeks of learning time. In grades 9-12, .05 standard deviation 
translates to approximately 11 and 12 weeks’ worth of total achievement gain in standardized math and English 
language arts. 


1 The priority innovations included in the analyses were identified by Georgia Department of Education staff as the most policy-relevant 
innovations to study from the much larger set of innovations included in the survey. 
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Table B4. Cohort and analysis structure for each performance contract adoption year 


Cohort structure Analysis structure 
Comparison 
Year of Comparison cohorts 
PNeloyohatoya) performance Change in Years included in cohorts contributing 
(oo) atolau contract achievement from meyer} pre-adoption contributing pre- post-adoption 
igeleye) Cohort adoption pre- to post-adoption cohorts id}aatem l-aleye| adoption data data 
Early 1 2008/09 Pre years—2009/10 1 2006/07- 2-9 na 
(N = 31) 2008/09 
2 2009/10 Pre years—2010/11 1-2 2006/07- 3-9 1 
2009/10 
3 2010/11 Pre years—2011/12 1-3 2006/07- 4-9 1-2 
———————— 2010/11 
4 2011/12 Pre years—2012/13 1-4 2006/07- 5-9 1-3 
2011/12 
5 2012/13 Pre years—2013/14 1-5 2006/07- 6-9 1-4 
2012/13 
6 2013/14 Pre years—2014/15 1-6 2006/07- 7-9 1-5 
2013/14 
7 2014/15 Pre years—2015/16 1-7 2006/07- 8-9 1-6 
2014/15 
Middle 8 2015/16 Pre years—2016/17 1-8 2006/07- 9 1-7 
(N = 61) 2015/16 
Late 9 2016/17 Pre years—2017/18 1-9 2009/10- na 1-8 
(N = 86) 2016/17 


na is not applicable. 
Note: Comparison cohorts listed contribute to analyses for focal cohorts other than their own. 
Source: Authors’ compilation based on information provided by the Georgia Department of Education. 


Qualitative analyses. The study team coded the interview transcripts in NVivo using the coding scheme outlined 
in box B1. 


Box B1. Coding scheme for interview data 


|. Background 
A. Description of your role in the district 
B. How long have you been in this role? 
C. What was your job before moving into this current role? 
ll. Types of innovations implemented 
A. Academic innovation 
1. Alternative education programming 
[FOR THIS AND EACH SUBSEQUENT INNOVATION] 
a. New innovation 
b. Expanding innovation 
c. Reason for choosing this innovation 
d. Waiver was needed 
e. Waiver was NOT needed 
. Classes for specific populations (gifted, English learner) 
. College and career academy 
. Community-based, internship, or work-based learning programs 
. Course scheduling 
. Credit requirements and availability 
. Dual enrollment 
. Early identification and intervention system 
. Enrichment or other specialty programs 
10. Graduation requirements 
11. Personalized instruction according to each student's needs 
12. Programs focused on improving college and career readiness 
13. Programs focused on improving graduation rates 
14. Summer programming 
15. Formative assessments to guide instruction 
B. Human resources innovations 
1. Additional summer academic programs for students 
. Beginning salary for hiring 
. Certification requirements 
. Differentiated salary systems 
. Instructional delivery model to change class size 
. Number of school days, daily school hours, or school year 
. Staff-to-student ratio in the classroom 
. Professional learning 
. Removing teacher tenure 
10. Teacher recruitment practices 
C. Financial innovations 
1. Instructional spending 
2. Noninstructional categorical allotments for instructional expenses 
3. Noninstructional spending 
4. State, local, and federal funds in support of school improvement plans 
Ill. Outcomes related to performance contract 
A. Student outcomes impacts 
[FOR THIS AND EACH SUBSEQUENT OUTCOME IMPACT] 
1. Specific innovations contributed to outcomes 
2. No specific innovation or outcome due to having a performance contract 
3. Methods of evaluating outcomes 
B. Staff outcomes impacts 


WO ONAN KWH 
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C. School administration practices and outcomes impacts 
D. Parental engagement practices impacts 
E. School climate outcomes impacts 
IV. Overall experience 
A. Why the district chose SWSS or Charter System 
1. Didn't want a governing body 
2. Wanted a governing body 
3. Not enough knowledge about charter option 
4. Uncomfortable with charter language 
5. Previous success 
6. Unique population needs 
B. Most beneficial aspect of performance contract 
1. Most beneficial aspect of governing council 
C. Supports for implementing innovations 
D. Challenges to implementing innovations 
1. Challenges to academics 
2. Challenges to human resources 
3. Challenges to finance 
E. If SWSS, who decided to apply for the waivers? 
F. Plans to expand 
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Appendix C. Supporting analyses 
This appendix details additional results and analyses that support the findings addressed in the study. 


Table C1. Results from final models estimating district-level change in student achievement in Georgia, 
2007/08-2017/18 


Grades 3-8 Grades 9-12 
Change in English Change in English Change in 
language arts Change in math language arts Algebra | 
NETar-ve)(- achievement achievement achievement achievement 
Post—performance contract adoption (pre— 0.02 (0.01) 0.03 (0.01)* —0.01 (0.01) —0.02 (0.02) 
performance contract adoption is reference 
group) 
District-level demographic characteristics of students and schools 
Percentage male (female is reference group) —0.01 (0.00)* —0.01 (0.00)* —0.01 (0.00)* —0.01 (0.00)* 
Race/ethnicity (White is reference group) 
Percentage of American Indian/Pacific —0.00 (0.01) —0.01 (0.01) —0.01 (0.01) 0.01 (0.01) 
Islander students 
Percentage of Asian students 0.01 (0.00)* 0.01 (0.00)* 0.00 (0.00)* 0.04 (0.00)* 
Percentage of Black students —0.00 (0.00)* —0.00 (0.00) —0.01 (0.00) —0.02 (0.00) 
Percentage of Hispanic students 0.00 (0.00) —0.00 (0.00)* —0.01 (0.00)* —0.01 (0.00)* 
Percentage of multiracial students 0.01 (0.00)* 0.01 (0.00)* —0.01 (0.00)* —0.00 (0.00) 
Percentage of students who ever had an —0.00 (0.00) —0.00 (0.00) —0.01 (0.00)* —0.02 (0.00)* 
individualized education program 
Percentage of students ever designated as English —0.00 (0.00)* —0.00 (0.00)* 0.00 (0.00) 0.00 (0.00) 
learner students 
Percentage of students ever eligible for free lunch —0.00 (0.00) 0.00 (0.00) —0.00 (0.00)* —0.00 (0.00) 
Percentage of students ever eligible for reduced- 0.01 (0.00)* 0.01 (0.00)* —0.00 (0.00) —0.00 (0.00) 
price lunch 
Grade level (grades 3 and 9 are reference group) 
Percentage of students in grade 4 0.00 (0.00) 0.00 (0.00) 
Percentage of students in grade 5 0.00 (0.00)* 0.00 (0.00) 
Percentage of students in grade 6 0.00 (0.00)* 0.00 (0.00) 
Percentage of students in grade 7 0.00 (0.00) 0.00 (0.00) 
Percentage of students in grade 8 0.00 (0.00)* 0.01 (0.00)* 
Percentage of students in grade 10 0.00 (0.00)* 0.01 (0.00)* 
Percentage of students in grade 11 —0.00 (0.00) 0.01 (0.00)* 
Percentage of students in grade 12 0.01 (0.00)* 0.01 (0.00) 
School type (traditional public is reference group) 
Percentage of Title | schools 0.00 (0.00) 0.00 (0.00)* 0.00 (0.00) 0.00 (0.00) 
Percentage of charter schools —0.00 (0.00)* —0.00 (0.00) —0.00 (0.00) 0.00 (0.00) 
Percentage of alternative schools 0.00 (0.00)* 0.00 (0.00)* 0.00 (0.00) 0.00 (0.00) 
Percentage of special education schools —0.00 (0.00) —0.00 (0.00)* 0.00 (0.00) 0.00 (0.00) 
Percentage of other schools —0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 
Percentage of magnet schools —0.00 (0.00)* —0.00 (0.00) —0.00 (0.00) 
Intercept 0.26 (0.18) —0.15 (0.20) 0.57 (0.07)* 0.91 (0.12)* 
Standard deviation of post—performance contract 0.09 (0.01)* 0.08 (0.01)* 0.08 (0.01)* 0.12 (0.02)* 
adoption effect 
Number of districts 178 178 178 178 
District-by-year observations 2,136 2,136 2,124 1,863 


* Statistically significant at the .05 level or lower. 

Note: Analysis also included district and school year fixed effects, not presented here. 

Source: Authors’ analysis of administrative and survey data from the Georgia Department of Education and data from the U.S. Department of Education’s 
Common Core of Data. 
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Table C2. Results from final models estimating district-level change in student achievement in Georgia and 
their relationship with district characteristics, 2007/08—2017/18 
(CT g-lo (=o es) 

Change in English 


Grades 9-12 


Change in 
Algebra | 
achievement 


language arts 
achievement 


Change in math 
achievement 


Change in English 


WEeTarle) (=) achievement 


Post—performance contract adoption (pre— 0.05 (0.03) 0.03 (0.03) 0.01 (0.03) 0.03 (0.04) 
performance contract adoption is 
reference group) 
District-level demographic characteristics of students and schools 
Percentage male (female is reference group) —0.01 (0.00)* —0.01 (0.00)* —0.01 (0.00)* —0.01 (0.00)* 
Race/ethnicity (White is reference group) 
Percentage of American Indian/Pacific —0.00 (0.01) —0.01 (0.01) —0.01 (0.01) 0.01 (0.01) 
Islander students 
Percentage of Asian students 0.01 (0.00)* 0.01 (0.00)* 0.00 (0.00) 0.04 (0.00)* 
Percentage of Black students —0.00 (0.00)* —0.00 (0.00) —0.01 (0.00)* —0.02 (0.00)* 
Percentage of Hispanic students 0.00 (0.00) —0.00 (0.00)* —0.01 (0.00)* —0.01 (0.00)* 
Percentage of multiracial students 0.01 (0.00)* 0.01 (0.00)* —0.01 (0.00)* —0.00 (0.00) 
Percentage of students who ever had an —0.00 (0.00) —0.00 (0.00) —0.01 (0.00)* —0.02 (0.00)* 
individualized education program 
Percentage of students ever designated as —0.00 (0.00)* —0.00 (0.00)* 0.00 (0.00) 0.00 (0.00) 
English learner students 
Percentage of students ever eligible for free —0.00 (0.00) 0.00 (0.00) —0.00 (0.00)* —0.00 (0.00) 
lunch 
Percentage of students ever eligible for 0.01 (0.00)* 0.01 (0.00)* —0.00 (0.00) —0.00 (0.00) 
reduced-price lunch 
Grade level (grades 3 and 9 are reference group) 
Percentage of students in grade 4 0.00 (0.00) 0.00 (0.00) 
Percentage of students in grade 5 0.00 (0.00)* 0.00 (0.00) 
Percentage of students in grade 6 0.00 (0.00)* 0.00 (0.00) 
Percentage of students in grade 7 0.00 (0.00) 0.00 (0.00) 
Percentage of students in grade 8 0.00 (0.00) 0.01 (0.00)* 
Percentage of students in grade 10 0.00 (0.00)* 0.01 (0.00)* 
Percentage of students in grade 11 —0.00 (0.00) 0.01 (0.00)* 
Percentage of students in grade 12 0.01 (0.00)* 0.01 (0.00) 
School type (traditional public is reference group) 
Percentage of Title | schools 0.00 (0.00) 0.00 (0.00)* 0.00 (0.00) 0.00 (0.00) 
Percentage of charter schools —0.00 (0.00)* —0.00 (0.00) —0.00 (0.00) 0.00 (0.00) 
Percentage of alternative schools 0.00 (0.00)* 0.00 (0.00)* 0.00 (0.00) 0.00 (0.00) 
Percentage of special education schools —0.00 (0.00) —0.00 (0.00)* 0.00 (0.00) 0.01 (0.00) 
Percentage of other schools —0.00 (0.00) 0.00 (0.00) 0.00 (0.00) —0.00 (0.00) 
Percentage of magnet schools —0.00 (0.00)* —0.00 (0.00) —0.00 (0.00) —0.00 (0.00) 
District characteristics (interactions with post-performance contract adoption) 
Urbanicity (suburban is reference group) 
City —0.04 (0.03) —0.03 (0.03) —0.00 (0.03) —0.14 (0.05)* 
Rural —0.03 (0.02) 0.02 (0.02) —0.01 (0.02) 0.01 (0.04) 
Town —0.04 (0.03) 0.01 (0.03) —0.06 (0.03)* —0.07 (0.05) 
Adoption cohort group (early [2005/06-2014/15] is reference group) 
Middle (2015/16) —0.01 (0.03) —0.02 (0.03) —0.05 (0.03) —0.00 (0.05) 
Late (2016/17) 0.01 (0.03) —0.02 (0.03) 0.00 (0.03) 0.04 (0.06) 
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Grades 3-8 
Change in English 


Grades 9-12 
Change in 
Algebra | 
achievement 


language arts 
achievement 


Change in math 
achievement 


Change in English 


WeeTare]e) (=) achievement 


Strategic Waivers School System (Charter 0.00 (0.03) 0.01 (0.03) 0.02 (0.03) —0.05 (0.05) 
System is reference group) 

Intercept 0.25 (0.18) —0.15 (0.20) 0.57 (0.07)* 0.92 (0.12)* 

Standard deviation of post—performance 0.09 (0.01)* 0.08 (0.01)* 0.07 (0.01)* 0.11 (0.02)* 
contract adoption effect 

Number of districts 178 178 178 178 

District-by-year observations 2,136 2,136 2,124 1,863 


* Statistically significant at the .05 level. 

Note: Analysis also included district and school year fixed effects, not presented here. 

Source: Authors’ analysis of administrative and survey data from the Georgia Department of Education and data from the U.S. Department of Education’s 
Common Core of Data. 


Table C3. Results from final models estimating district-level change in student achievement in Georgia and 
their relationship with district characteristics and prioritized innovations under performance contracts, 
2007/08-2017/18 


Grades 3-8 
Change in English 


Grades 9-12 


Change in 
Algebra | 
achievement 


language arts 
achievement 


Change in math 
achievement 


Change in English 


WeeTare}e) (=) achievement 


Post—performance contract adoption (pre— 
performance contract adoption is 
reference group) 


District-level demographic characteristics of students and schools 


Percentage male (female is reference group) 
Race/ethnicity (White is reference group) 
Percentage of American Indian/Pacific 
Islander students 
Percentage of Asian students 
Percentage of Black students 
Percentage of Hispanic students 
Percentage of multiracial students 
Percentage of students who ever had an 
individualized education program 
Percentage of students ever designated as 
English learner students 
Percentage of students ever eligible for free 
lunch 
Percentage of students ever eligible for 
reduced-price lunch 
Grade level (grades 3 and 9 are reference group) 
Percentage of students in grade 4 
Percentage of students in grade 5 
Percentage of students in grade 6 
Percentage of students in grade 7 
Percentage of students in grade 8 
Percentage of students in grade 10 
Percentage of students in grade 11 
Percentage of students in grade 12 
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0.03 (0.04) 


—0.01 (0.00)* 


0.02 (0.01) 


0.01 (0.00)* 
—0.00 (0.00) 
0.00 (0.00)* 
0.01 (0.00)* 
0.00 (0.00) 


—0.00 (0.00)* 
—0.00 (0.00) 


0.01 (0.00)* 


0.01 (0.00) 

0.01 (0.00)* 
0.01 (0.00)* 
0.01 (0.00)* 
0.01 (0.00)* 


0.03 (0.04) 


—0.01 (0.00)* 


0.00 (0.01) 


0.02 (0.00)* 
—0.00 (0.00) 
—0.00 (0.00)* 

0.02 (0.00)* 
—0.00 (0.00) 


—0.00 (0.00)* 
0.00 (0.00)* 


0.01 (0.00)* 


0.00 (0.00) 
0.01 (0.00)* 
0.00 (0.00) 
0.00 (0.00) 
0.01 (0.00)* 


0.01 (0.04) 


—0.01 (0.00)* 


—0.02 (0.01)* 


0.00 (0.00) 
—0.01 (0.00)* 
—0.00 (0.00)* 
—0.01 (0.00)* 
—0.01 (0.00)* 


0.00 (0.00) 
—0.00 (0.00) 


—0.00 (0.00) 


0.00 (0.00)* 
—0.01 (0.00) 
0.02 (0.00)* 


0.05 (0.07) 


—0.01 (0.00)* 


—0.01 (0.01) 


0.04 (0.00)* 
—0.02 (0.00)* 
—0.01 (0.00)* 

0.00 (0.00) 
—0.01 (0.00)* 


0.00 (0.00) 
—0.00 (0.00) 


0.00 (0.00) 


0.01 (0.00)* 
0.01 (0.00)* 
0.00 (0.00) 


C3 


Grades 3-8 Grades 9-12 


Change in English Change in 
language arts Change in math  Changein English Algebra | 
NFVar-le)(=1 achievement achievement achievement achievement 
School type (traditional public is reference group) 
Percentage of Title | schools 0.00 (0.00)* 0.00 (0.00)* 0.00 (0.00) —0.00 (0.00) 
Percentage of charter schools —0.00 (0.00)* —0.00 (0.00)* 0.00 (0.00) 0.00 (0.00) 
Percentage of alternative schools 0.00 (0.00)* 0.00 (0.00)* 0.00 (0.00) 0.00 (0.00) 
Percentage of special education schools —0.00 (0.00) —0.00 (0.00)* 0.00 (0.00) 0.01 (0.00) 
Percentage of other schools —0.00 (0.00) 0.00 (0.00) 0.00 (0.00) —0.00 (0.00) 
Percentage of magnet schools —0.00 (0.00)* —0.00 (0.00) —0.00 (0.00) —0.00 (0.00) 
District characteristics (interactions with post—performance contract adoption) 
Urbanicity (suburban is reference group) 
City —0.01 (0.04) 0.00 (0.04) 0.05 (0.04) —0.07 (0.07) 
Rural —0.02 (0.03) 0.01 (0.03) —0.01 (0.03) 0.02 (0.05) 
Town —0.02 (0.03) 0.02 (0.03) —0.05 (0.03) —0.04 (0.06) 
Adoption cohort group (early [2005/06—2014/15] is reference group, interactions with post-performance contract adoption) 
Middle (2015/16) 0.01 (0.04) —0.00 (0.04) —0.02 (0.04) —0.00 (0.06) 
Late (2016/17) 0.03 (0.04) —0.01 (0.04) 0.01 (0.04) —0.00 (0.07) 
Strategic Waivers School System (Charter 0.00 (0.03) 0.01 (0.04) 0.01 (0.03) —0.02 (0.06) 
System is reference) 
Academic innovations prioritized (interactions with post-performance contract adoption) 
Alternative education programming —0.03 (0.03) —0.02 (0.03) —0.07 (0.03)* —0.03 (0.05) 
College and career academy 0.00 (0.02) —0.03 (0.03) —0.03 (0.03) —0.05 (0.04) 
Personalized instruction 0.01 (0.04) 0.02 (0.04) 0.01 (0.04) 0.08 (0.08) 
School climate management 0.01 (0.18) 0.20 (0.20) 0.22 (0.22) —0.12 (0.38) 
Human resources innovations prioritized (interactions with post—performance contract adoption) 
Certification requirements —0.02 (0.02) —0.02 (0.02) 0.04 (0.03) —0.01 (0.04) 
Differentiated salary system 0.03 (0.03) 0.07 (0.03)* 0.04 (0.03) 0.06 (0.06) 
Class size 0.01 (0.02) 0.03 (0.02) —0.00 (0.02) —0.01 (0.04) 
Professional learning —0.05 (0.03) —0.02 (0.03) 0.02 (0.03) 0.02 (0.06) 
Financial innovations prioritized (interactions with post-performance contract adoption) 
Instructional spending 0.01 (0.02) —0.00 (0.03) —0.00 (0.03) —0.03 (0.04) 
Noninstructional spending 0.02 (0.02) 0.00 (0.03) 0.01 (0.03) —0.02 (0.04) 
Intercept 0.07 (0.21) —0.45 (0.24) 0.56 (0.08)* 0.97 (0.13)* 
Standard deviation of post—performance 0.09 (0.01)* 0.08 (0.01)* 0.08 (0.01)* 0.11 (0.02)* 
contract adoption effect 
Number of districts 133 133 133 133 
District-by-year observations 1,596 1,596 1,584 1,395 


* Statistically significant at the .05 level. 

Note: Analysis also included district and school year fixed effects, not presented here. 

Source: Authors’ analysis of administrative and survey data from the Georgia Department of Education and data from the U.S. Department of Education’s 
Common Core of Data. 
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Figure C1. Full list of innovative practices related to academic programs that Georgia districts prioritized for 
implementation after adopting a performance contract, 2019 


0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 


College and career academies 


Dual enrollment 
Alternative career pathways or industry certifications 


Alternative education programming 


Course scheduling 
Formative assessments 
College and career readiness programs 


Early identification and intervention system 


Online or blended curricula 

Personalized learning 

Community-based, internship, or work-based learning 
Enrichment or other specialty classes 

Credit requirements and availability 

Classes for specific populations 

Programs focused on improving graduation rates 
Graduation requirements 

Use of data to identify early intervention needs 
School climate management program 
Personalized instruction 

Core curricula 

Soft skills for career readiness 

Student promotion policies or practices 

Student retention policies or practices 

Summer programs 

Curriculum vendors 

Remedial education programs or student eligibility 
Bullying prevention programs 

School attendance policies or practices 

Special education programs or modified procedures 
School counseling services 

Student discipline policies or procedures 
Internships or work study opportunities 


School climate data collection 


oem 


Student placement policies or practices 


Textbooks 


MTotal responses m#1Rankings m#2Rankings m#3 Rankings 


Note: Sample includes 133 districts. 
Source: Authors’ analysis using survey data from the Georgia Department of Education. 
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Figure C2. Full list of innovative practices related to human resources that Georgia districts prioritized for 
implementation after adopting a performance contract, 2019 
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Certification requirements 

Staff-to-student ratio in the classroom 

Instructional delivery model to change class size 
Number of school days, daily school hours, or school year 
Professional learning approach 

Teacher recruitment practices 

Additional summer academic programs for students 
Differentiated salary system 

Type of staff providing instruction 

Supplemental programs for low-performing students 
Beginning salary for hiring 

Number of guidance counselors 

Staff duties or staff assignment practices 

Staff retention practices 

Parent engagement and roles 

Changed before-/after-school programs 

Increased before-/after-school programs 
Performance-based systems for staff 
Paraprofessionals in the classroom 


Staff evaluation and reward systems 


et 


School transportation availability 


Career ladder 


mTotal responses m#1Rankings m#2Rankings #3 Rankings 


Note: Sample includes 133 districts. 
Source: Authors’ analysis using survey data from the Georgia Department of Education. 
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Figure C3. Full list of innovative practices related to finances that Georgia districts prioritized for 
implementation after adopting a performance contract, 2019 
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Instructional spending 


State, local, and federal funds in support of school 
improvement plans 


Noninstructional spending 


Noninstructional categorical allotments for instructional 
expenses 


Categorical allotment for facilities or maintenance 


Instructional categorical allotments for noninstructional 
expenses 


Quality Basic Education funding for postsecondary programs 


Method to determine enrollment 


aa 


mTotal responses m#1Rankings m#2Rankings #3 Rankings 


Note: Sample includes 133 districts. 
Source: Authors’ analysis using survey data from the Georgia Department of Education. 


REL 2021-051 C-7 


Appendix D. Survey instrument 


This appendix presents the survey instrument used to collect data from Georgia district leaders. 


Thank you for volunteering to respond to this questionnaire. Your responses will provide insight on how Strategic 
Waiver School Systems (SWSS) or Charter Systems in Georgia use waiver flexibilities to implement innovative 
strategies. Your responses will be sent directly to the Regional Educational Laboratory (REL) Southeast research 
team and will be used for program improvement purposes only. All responses that relate to or describe identifiable 
characteristics of individuals may only be used for statistical purposes and will be kept confidential. No identifiable 
information will be shared outside of the REL Southeast research team. This questionnaire should take no more 
than 20 minutes to complete. 


Background information 


Question 1. Please select your district.* [Asterisks indicate a response was required.] 


Question 2. Please select your position in your district. (If you hold more than one position, please select your 
main role.)* 


Superintendent 

Assistant superintendent 

Chief financial officer/finance manager 
School improvement administrator 
Local governance coordinator 
Other—Write In (Required): 


oO oO: oO OO © 


Question 3. Are you considering changing from a SWSS to a Charter System, or from a Charter System to a SWSS?* 


o Yes 
o No 


Changes in practice 


Please share about innovations your district has implemented or may be considering since the approval of your 
SWSS or Charter System contract. 


For questions 4-6, please follow the steps included with each question. 


Academic program flexibility 
Question 4. Since approval of your contract, your district may have implemented innovations or may be 


considering innovations to initiate new academic programs, expand existing academic programs, and/or make 
other changes to academic programs. 


Please read through this list and select the top three academic program innovations that your district has 
implemented or plans to implement. 


If your district implemented other high priority innovations related to academic programs that are not listed 
below, you can list and describe them later in the questionnaire under question 8. 


Alternate career pathways and/or industry certifications 

Alternative education programming 

Bullying prevention program 

Classes for specific populations, such as gifted education and English learner 


oO 0 0 0 
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College and career academy 

Community-based, internship, or work-based learning programs 
Core curricula 

Course scheduling 

Credit requirements and availability 

Curriculum vendors 

Dual enrollment 

Early identification and intervention system 

Enrichment or other specialty classes 

Formative assessments used to guide instruction 
Graduation requirements 

Internships or work study opportunities 

Online and/or blended curricula 

Personalized instruction according to each student's needs 
Personalized learning for students 

Programs focused on improving college and career readiness 
Programs focused on improving graduation rates 
Remedial education programming or student eligibility 
School attendance policies or practices 

School climate data collection 

School climate management program 

School counseling services 

Soft skills for career readiness 

Special education programming or modified IEP procedures 
Student discipline policies or procedures 

Student placement policies or practices 

Student promotion policies or practices 

Student retention policies or practices 

Summer programs 

Textbooks 

Use of data to identify early intervention needs 
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Question 4, Follow-up #1. In the table below, please indicate whether the innovations you ranked above were to 
add new academic programs, expand existing academic programs, and/or make other changes to academic 
programs. 


Expand Other 
New existing changes 


First priority selected 
Second priority selected 
Third priority selected 


Question 4, Follow-up #2. In the table below, please indicate whether a waiver was or will be necessary for your 
district to implement each of your prioritized academic program innovations. 


Need waiver (Y/N) 


First priority selected 
Second priority selected 
Third priority selected 
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Human resources flexibility 


Question 5. Since approval of your contract, your district may have implemented innovations or may be 
considering innovations to expand, restructure, and/or make other changes to human resources. 


Please read through this list and select the top three human resource innovations that your district has 
implemented or plans to implement. 


If your district implemented other high-priority innovations related to human resources that are not listed below, 
you can list and describe them later in the questionnaire under question 8. 


Additional summer academic programs for students 
Beginning salary for hiring 

Career ladder 

Certification requirements 

Changed the structure or content of before- and/or after-school programs 
Differentiated salary system 

Increased before- and/or after-school programs 
Instructional delivery model to change class size 
Number of guidance counselors 

Number of school days, daily school hours, or school year 
Paraprofessionals in classroom 

Parent engagement and roles 

Performance-based systems for staff 

Professional learning approach 

School transportation availability 

Staff duties or staff assignment practices 

Staff evaluation and reward systems 

Staff retention practices 

Staff-to-student ratio in classroom 

Supplemental programs for low-performing students 
Teacher recruitment practices 

Type of staff providing instruction 
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Question 5, Follow-up #1. In the table below, please indicate whether the innovations you ranked above were to 
expand human resources, restructure existing human resources, and/or make other changes to human resources. 


Restructure Other 
Expand existing changes 


First priority selected 
Second priority selected 
Third priority selected 


Question 5, Follow-up #2. In the table below, please indicate whether a waiver was or will be necessary for your 
district to implement each of your prioritized human resource innovations. 


Need waiver (Y/N) 


First priority selected 
Second priority selected 
Third priority selected 
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Financial flexibility 

Question 6. Your district may have implemented innovations or may be considering innovations to expand your 
current funding, restructure your financial allocations and/or practices, and/or make other changes related to 
finances. 


Please read through this list and select the top three financial innovations that your district has implemented or 
plans to implement. 


If your district implemented other high priority innovations related to finances that are not listed below, you can 
list and describe them later in the questionnaire under question 8. 


Categorical allotment for facilities or maintenance 

Instructional categorical allotments for noninstructional expenses 
Instructional spending 

Method to determine enrollment 

Noninstructional categorical allotments for instructional expenses 
Noninstructional spending 

Quality Basic Education funding for postsecondary programs 

State, local, and federal funds in support of school improvement plans 
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Question 6, Follow-up #1. In the table below, please indicate whether the innovations you ranked above were to 
expand funding, restructure existing financial allocations, and/or make other changes to financial resources. 


Restructure Other 
Expand existing changes 


First priority selected 
Second priority selected 
Third priority selected 


Question 6, Follow-up #2. In the table below, please indicate whether a waiver was or will be necessary for your 
district to implement each of your prioritized financial innovations. 


Need waiver (Y/N) 


First priority selected 
Second priority selected 
Third priority selected 


Additional innovations 


Question 7. Since receiving approval of your SWSS or Charter System contract, has your district implemented, 
expanded, or changed in any way the use of the following innovations?* 


No change Implemented Expanded Reduced N/A 
Local school councils ° fe) ° fe) ° 
School choice request procedures fe) fe) fe) ° ° 


Question 8. Are there any other high-priority innovations related to academic programs, human resources, 
finances, or other flexibilities that your district implemented since becoming a SWSS or Charter System? If so, 
please list those innovations and in two to three sentences describe why they are a priority for your district. 


Perceived impact 


Thank you for indicating the innovations your district prioritized. In this section, you will have an opportunity to 
share how those innovations affected your district and/or schools. 


Please think about only the innovations you identified while answering questions 4-6 (listed below for your 
reference) and question 8: 
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e Your district's innovations related to academic programs: [insert innovations selected above] 
e Your district's innovations related to human resources: [insert innovations selected above] 
e Your district's innovations related to finances: [insert innovations selected above] 


Question 9. In your opinion, how have the following outcomes been affected by implementing the high-priority 
innovations you identified? For each outcome, choose one of the following: (1) improved substantially, (2) 
improved somewhat, (3) stayed the same, (4) worsened somewhat, (5) worsened substantially, (6) not relevant 


to our contract. 


Then, please list which one or more of those innovations you attribute to making the most change for each 


outcome.* 
Innovation(s) 
affecting the 
Outcome most change 


Student outcomes 


Academic achievement 


Attendance 


Grades 


Graduation rates 


Staff outcomes 


Teacher retention 


Principal retention 


Teachers’ instructional practices 


Teachers’ compensation 


Teachers’ opportunities to advance in their teaching profession 


Teachers’ opportunities to collaborate with colleagues at their school 


Teachers’ opportunities for professional development 


Support for new teachers 


Time available for collaboration 


Principals’ instructional leadership skills 


Support for new teachers 


Staff retention 


School budgets 


Ability to hire new teachers 


Maintenance of school buildings and facilities 


Parental engagement practices 


Parental engagement with their child’s schooling 


Parental engagement with programs aimed at improving students’ academic outcomes 


Parental engagement with developing school curricula 


Parental engagement with school policy decisions 


Parental engagement with planning career pathway for student 


Parental engagement with student supplemental services 


School climate outcomes 


Student behavior 


Perceptions of safety 


Relationships between teachers and students 


Relationships among teachers 


Relationships between school leaders and teachers 


Relationships between school leaders and their peers 


Question 10. Are the schools in your district implementing the innovations that you identified in questions 4—6 


and question 8?* 


o No, implementation of innovations is not yet happening at the school level. 
oO Yes, implementation of innovations is consistent across the schools in our district. 
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© Yes, implementation of innovations is different across the schools in our district. 
o Other—Please explain: 


Challenges 


Question 11. To what extent is each of the following a challenge to your efforts to implement innovations to meet 
your performance contract goals? 


Nota Minor Moderate Major 
challenge challenge challenge challenge 


Staff turnover 

Staff recruitment 

Available funding 

Parent support 

Community support 

Lack of understanding of possible innovations 
Insufficient staff to implement innovations 
Changes to state policies or procedures 


Stakeholders 


Question 12. Please select the three most influential stakeholders for the selection of innovations in your district 
and whether they had (1) a total influence, (2) a great deal of influence, or (3) a moderate amount of influence 
over the selection of innovations.* 


District administrators 

District staff (other than district administrators) 

District leaders in other districts (e.g., a cohort of district leaders) 
GaDOE staff 

State Board of Education 

School administrators in my district 

School staff (other than school administrators) 

School governance team 

School staff external to my district 

Parents 

Community members 

Local business community 

Institutions of higher education 

External organizations (e.g., the Charter System Foundation, EMO/CMO, Southeast Comprehensive Center) 
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Question 12, Follow-up: Please specify the external organization(s) if ranked. 


Value 


Question 13. What is the most beneficial aspect of your district’s performance contract?* 


Having performance targets/goals to meet 

Refocusing on improving student outcomes 

Avoiding consequences 

Making stakeholder involvement transparent 
Facilitating continuous improvement 

Changing culture from one of compliance to innovation 
Other—Write In (Required): 


Oo 0 OO OO O OO OC 
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Appendix E. Interview protocol 


This appendix presents the interview protocol used to collect interview data from Georgia district leaders. 


Introduction 

Hello, my name is [GaDOE researcher]. [If applicable:] With me today is [REL Southeast researcher’s name] from 
the Regional Educational Laboratory (REL) Southeast, our partner on this study. Thanks again for taking the time 
to speak with us this [morning/afternoon]. We anticipate the interview will last approximately 90 minutes. 


Before we start, do you have any questions about the background information on our study we sent you? As noted 
in that information, we’re interviewing district administrators about implementing waivers. Be assured this study 
is not to assess you or how well waivers are being implemented in your district. It is to learn about your 
experiences to help evaluate the effectiveness of the entire waiver program, consider future changes, and provide 
guidance on best practices for waiver implementation. 


We also sent you a consent form with more detailed information about this evaluation and your rights as a 
participant. As a reminder, your participation in this interview is completely voluntary and will not affect your 
employment. You may skip any question that you do not feel comfortable answering, and you may choose to 
leave the interview at any time. In order to participate, you will need to sign the first line of the consent form. Are 
you okay with this? Do you have any questions? 


Be assured there are no right or wrong answers; we are seeking what you know to be facts, and we are also 
seeking your opinions, so please share with us to the best of your ability. We hope to have a lively and frank 
conversation. We would like to capture all of the information you share as accurately as possible, so will you 
permit us to audio-record the interview? If there is anything you would like to have off the record, please let us 
know, and we will not record it. 


Please know that we will keep everything you tell us confidential. We will never use your name when we report 
our findings or repeat anything you say. Only our research team, who is responsible for analyzing the data and 
writing a report about our findings, and an external transcription company will have access to what you say. 


Before we begin, do you have any questions? 


Background questions 


Let’s begin with a brief introduction. 


e Please share your name and a brief description of your role in the district. 
e How long you have been in this role? 
e What was your job before moving into your current role? 


Decision making 

GaDOE’s motivation when they asked REL Southeast to conduct this study was to understand whether the 
Strategic Waiver Flexibility policy was a good policy decision statewide. In order to answer this question, we need 
to go back to the beginning of your district’s implementation of the policy. For that reason, we want you to think 
about the initial transition your district made from traditional district to [SWSS or Charter System] when answering 
questions during this interview. 


Our first interview section asks about your district’s decisionmaking processes related to applying for [SWSS or 
Charter System] status in general [and applying for specific waivers if SWSS] when your district first made the 
transition. 
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e Why did your district decide to apply for [SWSS or Charter System status] in general? 


© Prompts for further information (if not answered above): 
o Who was involved in the decision-making process? Briefly describe the decision-making process. 
o Why did you choose to become a [SWSS or Charter System] rather than [if SWSS above, then Charter 
System here or vice versa]? 


[If survey response indicates they plan to switch from a SWSS to Charter System or vice versa]: Why did you decide 
to change the district flexibility option? 


[lf SWSS]: Who made the decision to apply for your district’s waivers? What factors did they consider? 


[lf SWSS]: We would like to ask you a few general questions about your district waiver priorities when you first 
contracted to become an SWSS. 


e What were the top two waivers that were a priority for your district when you first became an SWSS? 
e Are those two waivers still the top two priority waivers for your district? 
e If not, what are the top two waivers that are a priority for your district now? 


Waiver/SWSS implementation 

Let’s now focus on the approved waivers in your district’s initial [SWSS or Charter System] contract. Again, this 
section asks about your initial transition from a non-[SWSS or Charter System] to a [SWSS or Charter System]. 
Although your district may have changed its performance contract since first converting to a [SWSS or Charter 
System], please think about your initial contract when answering the following questions. 


e Has your district implemented all of the waivers in your initial contract? 
e Which waiver did your district implement first? Why was that waiver implemented first? 
e Are there any waivers your district requested that you will not be implementing? If so, for what reason? 


Innovations related to academic programs 

On the questionnaire, when asked about academic programs, you indicated that the district prioritized [top two 
innovations] upon receiving [SWSS or Charter System] status. | have a few questions about each of these 
innovations. 


e How was [the first priority innovation] selected as a priority to implement in your district? 

e Who were the key personnel in selecting this innovation to implement and how did they influence selection? 
(for example, district administrators, other district staff, GaDOE staff, school administrators in your district, 
parents, community members, groups or organizations external to your district) 

e How did the district implement [the first priority innovation]? 

e Who were/are key personnel in implementing [the first priority innovation]? 

e How did the district communicate about this implementation of [the first priority innovation] with your 
schools? 

e What supports were in place or were needed to implement [the first priority innovation]? 

e What were barriers to implementation? 

e Since implementing [the first priority innovation], did your district modify this innovation or make additional 
decisions about this innovation? 

e Which waiver was necessary for your district to implement [the first priority innovation]? \f none, why not? 


Now, let’s talk about the implementation of [the second priority innovation related to academic programs]. 


e How was [the second priority innovation] selected as a priority to implement in your district? 
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Who were the key personnel in selecting [the second priority innovation] to implement and how did they 
influence selection (for example, district administrators, other district staff, GaDOE staff, school 
administrators in your district, parents, community members, groups or organizations external to your 
district)? 

Please describe the activities/actions your district has taken to implement [the second priority innovation]. 
Who were/are key personnel in implementing [the second priority innovation]? 

How did the district communicate about this implementation of [the second priority innovation] with your 
schools? 

What supports were in place or were needed to implement [the second priority innovation]? 

What were barriers to implementation? 

Since implementing [the second priority innovation], did your district modify this innovation or make 
additional decisions about this innovation? 

Which waiver was necessary for your district to implement [the second priority innovation]? |f none, why not? 


Innovations related to human resources 


On the survey, when asked about human resources, you indicated that the district prioritized [top two innovations] 
upon receiving [SWSS or Charter System] status. | have a few questions about each of these innovations. 


How was [the first priority innovation] selected as a priority to implement in your district? 

Who were the key personnel in selecting [the first priority innovation] to implement and how did they 
influence selection? (for example, district administrators, other district staff, GaDOE staff, school 
administrators in your district, parents, community members, groups or organizations external to your 
district) 

How did the district implement [the first priority innovation]? 

Who were/are key personnel in implementing [the first priority innovation]? 

How did the district communicate about this implementation of [the first priority innovation] with your 
schools? 

What supports were in place or were needed to implement /the first priority innovation]? 

What were barriers to implementation? 

Since implementing [the first priority innovation], did your district modify this innovation or make additional 
decisions about this innovation? 

Which waiver was necessary for your district to implement [the first priority innovation]? If none, why not? 


Now, let’s talk about the implementation of [the second priority innovation related to academic programs]. 


How was [the second priority innovation] selected as a priority to implement in your district? 

Who were the key personnel in selecting [the second priority innovation] to implement and how did they 
influence selection (for example, district administrators, other district staff, GaDOE staff, school 
administrators in your district, parents, community members, groups or organizations external to your 
district)? 

Describe the activities/actions your district has taken to implement [the second priority innovation]. 

Who were/are key personnel in implementing [the second priority innovation]? 

How did the district communicate about this implementation of [the second priority innovation] with your 
schools? 

What supports were in place or were needed to implement [the second priority innovation]? 

What were barriers to implementation? 
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e Since implementing [the second priority innovation], did your district modify this innovation or make 
additional decisions about this innovation? 
e Which waiver was necessary for your district to implement [the second priority innovation]? If none, why not? 


Innovations related to finances 
On the survey, when asked about finances, you indicated that the district prioritized [top two innovations] upon 
receiving [SWSS or Charter System] status. | have a few questions about each of these innovations. 


e How was [the first priority innovation] selected as a priority to implement in your district? 

e Who were the key personnel in selecting [the first priority innovation] to implement and how did they 
influence selection (for example, district administrators, other district staff, GaDOE staff, school 
administrators in your district, parents, community members, groups or organizations external to your 
district)? 

e How did the district implement [the first priority innovation]? 

e Who were/are key personnel in implementing [the first priority innovation]? 

e How did the district communicate about this implementation of [the first priority innovation] with your 
schools? 

e What supports were in place or were needed to implement [the first priority innovation]? 

e What were barriers to implementation? 

e Since implementing [the first priority innovation], did your district modify this innovation or make additional 
decisions about this innovation? 

e Which waiver was necessary for your district to implement [the first priority innovation]? \f none, why not? 


Now, let’s talk about the implementation of [the second priority innovation related to academic programs]. 


e How was [the second priority innovation] selected as a priority to implement in your district? 

e Who were the key personnel in selecting [the second priority innovation] to implement and how did they 
influence selection (for example, district administrators, other district staff, GaDOE staff, school 
administrators in your district, parents, community members, groups or organizations external to your 
district)? 

e Describe the activities/actions your district has taken to implement [the second priority innovation]. 

e Who were/are key personnel in implementing [the second priority innovation]? 

e How did the district communicate about this implementation of [the second priority innovation] with your 
schools? 

e What supports were in place or were needed to implement [the second priority innovation]? 

e What were barriers to implementation? 

e Since implementing [the second priority innovation], did your district modify this innovation or make 
additional decisions about this innovation? 

e Which waiver was necessary for your district to implement [the second priority innovation]? If none, why not? 


Other innovations 
Did you implement any other high-priority innovations you would identify as being innovative compared to prior 
practices or to other districts in the state? 


e Prompts: use of local school councils, school choice request procedures 


Please describe, briefly, how each innovation was implemented (repeat questions about implementation for each 
innovation mentioned in response to this question—limit to two innovations). 
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Outcomes/changes 


You’ve now described innovations you have begun implementing or have been implementing for a while. In the 
next set of questions, please continue to think about your initial performance contract. We will ask you to describe 
whether and how implementing the waivers we just discussed has led to any changes in your district and schools 
in several areas, such as student outcomes, staff outcomes, human resources and financial outcomes, parental 
engagement outcomes, and school climate outcomes. 


e Of these kinds of outcomes (student outcomes, staff outcomes, school administration practices and 
outcomes, parental engagement outcomes, and school climate outcomes), which kinds of outcomes do you 
think have resulted because you’ve implemented waivers as a whole? 


Now, | want to ask you about each of these types of outcomes you identified in more detail. 

Student outcomes [if identified by respondent]: 

e In what ways did student outcomes change in your district as a result of implementing waivers as a whole? 
oO Prompts: academic achievement, attendance, graduation rates 


e Now, please think about the specific innovations you described earlier. Are there particular innovations that, 
in your opinion, contributed most to the changes in outcomes we just discussed? If so, how did those 
innovations contribute to changes in outcomes? (Prompt with changes they identified in response to the 
previous question.) 


Staff outcomes [if identified by respondent]: 
e In what ways did staffing outcomes change as a result of implementing waivers as a whole? 


oO Prompts: teachers’ instructional practices, teacher compensation, teachers’ opportunities to collaborate 
with colleagues at their school, teachers’ opportunities for professional development and advancement, 
support for new teachers, principals’ instructional leadership, principal and teacher retention 


e Now, please think about the specific innovations you described earlier. Are there particular innovations that, 
in your opinion, contributed most to the changes in outcomes you just described? If yes, how did those 
innovations contribute to changes in outcomes? (Prompt with changes they identified in response to the 
previous question.) 


School administration practices and outcomes [if identified by respondent]: 


e In what ways have school administration practices and outcomes changed as a result of implementing waivers 
as a whole? 


oO Prompts: school budgets, ability to hire new teachers, maintenance of school buildings and facilities 


e Now, please think about the specific innovations you described earlier. Are there particular innovations that, 
in your opinion, contributed most to the changes in outcomes we just discussed? If so, how did those 
innovations contribute to changes in outcomes? (Prompt with changes they identified in response to the 
previous question.) 


Parental engagement outcomes [if identified by respondent]: 
e In what ways has parent engagement in your district changed as a result of implementing waivers as a whole? 


oO Prompts: parent engagement with their child’s schooling, parent engagement with programs aimed at 
improving students’ academic outcomes, parent engagement with developing school curricula, parent 
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engagement with school policy decisions, parent engagement with planning career pathway for student, 
parent engagement with student supplemental services 


e Now, please think about the specific innovations you described earlier. Are there particular innovations that, 
in your opinion, contributed most to the changes in outcomes we just discussed? If so, how did those 
innovations contribute to changes in outcomes? (Prompt with changes they identified in response to the 
previous question.) 


School climate outcomes [if identified by respondent]: 
e In what ways has school climate in your schools changed as a result of implementing waivers as a whole? 


oO Prompts: student behavior, perceptions of safety, relationships between teachers and students, 
relationships among teachers, relationships between teachers and school leaders, relationships among 
school leaders and their peers 


e Now, please think about the specific innovations you described earlier. Are there particular innovations that, 
in your opinion, contributed most to the changes in outcomes you just described? If yes, how did those 
innovations contribute to changes in outcomes? (Prompt with changes they identified in response to the 
previous question.) 


Other changes 


e Are there other outcomes that have changed in your district as a result of implementing waivers as a whole? 


oO If so, please think about the specific innovations you implemented. Are there particular innovations that, 
in your opinion, contributed most to the changes in outcomes we just discussed? If so, how did those 
innovations contribute to changes in outcomes? (Prompt with changes they identified in response to the 
previous question.) 


e To conclude our discussion of outcomes, what do you believe has contributed most to changes in outcomes 
as a whole? 


Evaluating implementation outcomes 

We know that evaluation of whether your district meets its performance targets is part of your performance 
contract. Beyond this evaluation, we are interested in understanding whether and how your district evaluates 
how innovations are being implemented and how the practices involved in implementation are related to 
outcomes of interest. 


e Do you have any processes in place for evaluating how waiver implementation is linked to changes in 
outcomes? 


© If so, please describe the process(es). 
© If not, do you have plans to begin monitoring the implementation process and how implementation is 
linked to outcomes? 


Future plans, supports, and challenges 


In this final section, we want you to reflect about the entire concept of waivers as a whole, how waivers have 
worked for your district, and how you plan to move forward with waivers in the future. 


e = Inthinking about implementing innovations as a result of becoming a [SWSS or Charter System], what state-, 
district-, or school-level supports have facilitated implementation of your performance contract? 
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What state-, district-, or school-level challenges have you encountered in your efforts to implement 
innovations? 


oO Prompts: staff turnover, available funding, parent support, community support, lack of understanding of 
possible innovations, limited time, insufficient staff, changes to state policies or procedures, external 
intervention 


Do you have plans to implement additional high-priority innovations or expand any current innovations you 
are implementing related to any of your waivers? 
If yes, 


oO What is your anticipated timeline for implementation? 

oO What outcomes do you anticipate as a result of these planned innovations? 

© How do you plan to monitor and evaluate implementation? 

© What supports and/or resources will help your [SWSS or Charter System] to implement these innovations? 
oO What challenges pose a barrier to implementation? 


[If applicable—if the district has renewed their contract:] How was your district’s performance contract 
renewal affected by your initial implementation experiences? 
What does your district see as the most beneficial aspect of having a performance contract with the state? 


Is there anything we haven’t discussed that you think | should know about your district’s performance contract 


and its implementation? 
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